Data-driven Web Design

This short paper summarizes challenges and opportunities of applying machine learning methods to Web design problems, and describes how structured prediction, deep learning, and probabilistic program induction can enable useful interactions for designers. We intend for these techniques to foster new work in data-driven Web design. 1. Machine Learning for Web Design The Web provides an enormous repository of design knowledge: every page represents a concrete example of human creativity and aesthetics. Given the ready availability of Web data, how can we leverage it to help designers? This note outlines three machine learning applications which enable new interaction mechanisms for Web design: structured prediction for rapid retargeting, deep learning for design-based search, and probabilistic program induction for operationalizing design patterns. All of these techniques leverage structure that is intrinsic to Web designs. In machine learning applications, working with structured representations affords significant advantages over unstructured data sets, such as images or text (Socher et al., 2011). On the Web, every page is associated with a Document Object Model (DOM) tree, which can be used along with render-time information to bootstrap a visual information hierarchy for designs. Appearing in Proceedings of the 29 th International Conference on Machine Learning, Edinburgh, Scotland, UK, 2012. Copyright 2012 by the author(s)/owner(s). Figure 1. Our example-based retargeting algorithm, Bricolage, automatically renders pages in new layouts and styles. To apply machine learning techniques to Web design, we need to collect a corpus of training examples. Although traditional Web crawlers make it easy to scrape content from pages, acquiring and managing all the resources necessary to preserve a page’s render-time appearance is much more difficult. Furthermore, with the advent of clientand server-side scripting and dynamic HTML, many modern Web pages are mutable and may change between accesses, frustrating algorithms that expect consistent training data. To overcome these challenges, we have constructed a new kind of Web repository. The repository is populated via a bespoke Web crawler, which requests pages through a caching proxy backed by an SQL database. As a page is crawled, all requested resources are versioned and stored, its DOM tree is processed to produce a static visual hierarchy of the page’s structure, and a set of semantic and vision-based features are calculated on each constituent page component. These structures are then exposed through a RESTful API, allowing fast component-wise queries on features. We Data-driven Web Design have found that this design repository enables the rapid development of a diverse set of machine learning applications that support creative work. 2. Structured Prediction People frequently rely on templates when designing Web sites. While templates provide a simple mechanism for rendering content in different layouts, their rigidity often limits customization and yields cookiecutter designs. Bricolage is a structured prediction algorithm that allows any page on the Web to serve as a design template (Kumar et al., 2011a). The algorithm works by matching visually and semantically similar elements in pages to create coherent mappings between them. Once constructed, these mappings are used to automatically transfer the content from one page into the style and layout of another (Figure 1). Bricolage learns correspondences between pages by training on a set of human mappings gathered via crowdsourcing. It learns how to flexibly optimize visual, semantic, and structural considerations using the generalized perceptron algorithm (Kumar et al., 2011b). Our experiments show that flexibly preserving structure is essential for predicting human-like mappings. The method enables a diverse set of design interactions, including rapid prototyping, retargeting between form factors, and measuring the similarity of Web designs.